Convexity, Surrogate Functions and Iterative Optimization in Multi-class Logistic Regression Models
نویسندگان
چکیده
Abstract. In this paper, we propose a family of surrogate maximization (SM) algorithms for multi-class logistic regression models (also called conditional exponential models). An SM algorithm aims at turning an otherwise intractable maximization problem into a tractable one by iterating two steps. The S-step computes a tractable surrogate function to substitute the original objective function, and the M-step seeks to maximize this surrogate function. We apply SM algorithms to logistic regression models, leading to the standard SM, generalized SM, gradient SM, and quadratic SM algorithms. Compared with Newton’s method, these SM algorithms dramatically save computational costs when either the dimensionality or number of data samples is huge. Finally, we demonstrate the efficacy of these SM algorithms and compare their empirical performance on text categorization.
منابع مشابه
INTRODUCTION AND DEVELOPMENT OF SURROGATE MANAGEMENT FRAMEWORK FOR SOLVING OPTIMIZATION PROBLEMS
In this paper, we have outlined the surrogate management framework for optimization of expensive functions. An initial simple iterative method which we call the “Strawman” method illustrates how surrogates can be incorporated into optimization to stand in for the most expensive function. These ideas are made rigorous by incorporating them into the framework of pattern search methods. The SMF al...
متن کاملGAP Safe screening rules for sparse multi-task and multi-class models
High dimensional regression benefits from sparsity promoting regularizations. Screening rules leverage the known sparsity of the solution by ignoring some variables in the optimization, hence speeding up solvers. When the procedure is proven not to discard features wrongly the rules are said to be safe. In this paper we derive new safe rules for generalized linear models regularized with `1 and...
متن کاملTwo-temperature logistic regression based on the Tsallis divergence
We develop a variant of multiclass logistic regression that achieves three properties: i) We minimize a non-convex surrogate loss which makes the method robust to outliers, ii) our method allows transitioning between non-convex and convex losses by the choice of the parameters, iii) the surrogate loss is Bayes consistent, even in the non-convex case. The algorithm has one weight vector per clas...
متن کاملLogistic Regression: Tight Bounds for Stochastic and Online Optimization
The logistic loss function is often advocated in machine learning and statistics as a smooth and strictly convex surrogate for the 0-1 loss. We investigate the question of whether these smoothness and convexity properties make the logistic loss preferable to other widely considered options such as the hinge loss. We show that in contrast to known asymptotic bounds, as long as the number of pred...
متن کاملDecentralized Online Learning with Kernels
We consider multi-agent stochastic optimization problems over reproducing kernel Hilbert spaces (RKHS). In this setting, a network of interconnected agents aims to learn decision functions, i.e., nonlinear statistical models, that are optimal in terms of a global convex functional that aggregates data across the network, with only access to locally and sequentially observed samples. We propose ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2004